Improving Performance of Classification Models with Textual Data

The main objective in this study is to measure the effect of unstructured text on classification performance. A large dataset of aviation incidents reports was used in this study. In aviation incidents the proportion attributable to human factors is close to 90%. Therefore accurate identification of the presence of human factors in past aviation incidents is critical to improving aviation safety. Most existing research for identifying the causes of aviation incidents are based on manual work by experts and often rely on just structured data. In this study, a text mining approach was used to extract topics from the text narratives in incident reports to detect the presence of human factors in the incidents. These topics, along with structured data in the incident reports, were then used to build classifiers using three different models: memory-based reasoning (MBR), support vector machines (SVM), and decision trees (DT). The preliminary results from the experiments show that the best model in terms of the overall classification rate is the DT model with textual and structured data. The best model in terms of sensitivity is SVM with textual and structured data. These results show that the addition of textual data can potentially improve classification performance.